Presentation from the international conference on soft computing and soft engineering

1. SENTIMENT ANALYSIS OF TWEETS
Predicting a Movie's Box Office Success
Vasu Jain

2. AGENDA
1. Introduction
2. Related Work
3. Methodology
4. Experiments
5. Conclusion
6. Quse/Ans
Image source: SNLP Slides for Sentiment Analysis

3. INTRODUCTION
About Twitter
• Social networking and microblogging service
• Enables users to send and read messages
• Messages of length up to 140 characters, known as "tweets".
Tweets contain rich information about people’s preferences.
People share their thoughts about movies using Twitter.
Data analysis on twitter data to predict the success of a movie.

4. INTRODUCTION
People’s opinions towards a movie have huge impact on its
success.
Our project includes prediction using Twitter data, and analysis of
the prediction results.
High volume of positive tweets may indicate success of a movie.
But how to quantify ?
Image source: http://www.demainlaveille.fr/2012/05/06/pourquoi-twitter-ne-peut-pas-predire-les-elections-presidentielles/

6. RELATED WORK
Using social media to predict the future becomes very popular in recent
years.
• Predicting the Future with Social Media (Sitaram Asur & Bernardo A.
Huberman, 2010) tries to show that twitter-based prediction of box
office revenue performs better than market-based prediction.
• Predicting IMDB movie ratings using social media (Andrei Oghina,
Mathias Breuss, Manos Tsagkias & Maarten de Rijke 2012) uses twitter
and youtube data to predict the imdb scores.
Our project includes prediction using Twitter data and investigation on two
new topics based on the prediction results.

7. RELATED WORK
• Predicting the results of presidential election (USC Annenberg
Innovation Lab & USC SAIL).
• Sentiment 140 to discover the Twitter sentiment (sentiment140.com) .
No movie prediction is provided.

8. OUR WORK
• Data Collection: existing twitter data set and recent tweets via
Twitter API
• Data Pre-processing: get the "clean" data and transform it to the
format we need
• Sentiment Analysis: train a classifier to classify the tweets as:
positive, negative, neutral and irrelevant
• Prediction: use the statistics of the tweets' labels to predict the
movie success (hit/flop/average)

9. METHODOLOGIES: Data Collection & Crawling
2009 Data set Subset of Stanford dataset (now unavailable)
• 477 Million Tweets, period of June – Dec 2009
• Filtered tweets during critical period for movie
• 68.7 GB datasets (compressed format)
• 30 movies, 6 Million relevant Tweets
2012 Data set live crawling using a script
• Streaming API of python library for Twitter
to collect data
• Data Retrieval using keywords for movies
• Data collection focus on critical period
• 8 Movies, 2.5 Million Tweets
Image source: http://drupal.org/project/twitterminer

10. METHODOLOGIES: Data Collection & Crawling
160000
140000
120000
100000
80000
60000
40000
20000
0
week -6
week -5
week -4
week -3
week -2
week -1
week 0
week 1
week 2
week 3
week 4
week 5
week 6
week 7
week 8
week 9
week 10
week 11
week 12
week 13
week 14
week 15
week 16
week 17
week 18
week 19
week 20
week 21
week 22
week 23
week 24
Tweets Number
Critical Period for movie “Harry Potter and the Half-Blood Prince".
Show the relationship between sent time and number of tweets for the movie
Image source: http://drupal.org/project/twitterminer

11. METHODOLOGIES: Data Preprocessing
Why data preprocessing ?
• Lot of noisy, spam, irrelevant tweets in our dataset
• Convert the data to input format for our sentiment
analysis tools.
Techniques for preprocessing:
• Removing URLs, user handles
• Language detection to discard tweets not in English
• Split the dataset into small chunks ~25000 Tweets/Chunk
• Process chunks distributely
• Filter for tweets related to target movies using regular
expression.
Image source: http://mashable.com/2012/03/18/tweets-more-trustworthy-study/

12. METHODOLOGIES: Sentiment Analysis
Algorithm:
• Labelling tweets using Lingpipe sentiment analyzer, a natural
language processing toolkit.
• Sentence (tweet) based analysis with a logistic regression classifier.
(Accuracy up to 80%)
• Training & evaluation using 2009 dataset, testing on 2012 dataset.
• Trained classifier labels tweet as positive, negative, neutral or
irrelevant.
• Calculate PT-NT Ratio for every movie. PT-NT Ratio is a function
over parameters positive tweet ratio, negative tweet ratio, total
tweets, neutral tweets, irrelevant tweets.
• Thresholds to determine regions for PT-NT Ratio. Each region
corresponds to Hit, Flop, Average results for movies.
• Movie success correlated with PT-NT Ratio.

13. Experiments: Analysis of 30 Movies (Released in 2009)

14. Experiments: Movies vs. P/N Ratio, Profit Ratio

15. Experiments: Movies (Released in 2009) vs. PT-NT Ratio

16. Experiments: Analysis of 8 Movies (Released in 2012)

17. Experiments: Movies (Released in 2012) vs. PT-NT Ratio

18. Conclusion
Prediction for 2012 movies using our analysis:
5 movies: Hit
1 movie: Super hit
1 movie: Average business
Could not determine success rate for one due to it data unavailability.
Comparing our prediction results with box office results till date
Prediction: exactly right for four cases
On border line between hit and average for one case
For remaining movies we lack data to check our prediction onfidence .
Half accuracy score if movie’ s classification near border.
Score of 4.5 out of 5 for accuracy that is equal to 90%.
Great achievement for our model even though there were limitations with
number of movies, hand labeled tweets etc.

19. Future Work
Bottlenecks:
1. Twitter data crawled by third party.
2. Limitation with Twitter APIs for crawling data.
3. Noise included in randomly picked 200 tweets.
4. Movies being released in limited number of theaters
(Not enough data)
With more data, model can be more accurate and reliable.
Future work:
1. Using different other models and algorithms.
2. Temporal analysis can be added as a future work in the project.
3. Consideration of Retweets as a factor
Image source: http://www.theispot.com/whatsnew/2012/2/brucie-rosch-twitter-data.htm

20. Thank you
Ques/Ans